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16  Abc-roc-vr^g  report  describes  and  documents  Pass-Fail  procedures  for  the  new  PSS 
Training  Program.  New  types  of  measures  and  sets  of  nerms  are  used  to  create 
standards  of  performance  that  students  must  meet  to  become  eligible  for  acceptance 
into  the  operational  facility  work  force.  One  type  is  newly  developed  FSS  Skills 
Tests,  paper-and-pencil  simulations  of  laboratory  problems,  that  were  administered 
to  personnel  at  operational  Flight  Service  Stations  across  the  country  for  validation 
relation  to  on-the-job  performance,  and  standardization  purposes.  Other  types  are 
the  new  academic  block  tests  and  graded  laboratory  problems  created  for  each  of  the 
FSS  positions.  A  student  who  fails  both  the  graded  laboratory  problems »and  skills 
tests  in  the  Pilot  Briefing,  Inflight,  or  Emergency  Services  positions  is  subject  to 
failure.  A  student  who  fails  to  achieve  a  passing  score  on  a  final  Phase  Grade  also 
is  subject  to  failure.  The  Phase  Grade  is  a  weighted  average  of  performances  on  the 
academic  block  tests,  graded  laboratory  problems# (all  positions),  and  the  Pilot 
Briefing,  Inflight  and  Emergency 'Services  Skills  Tests.  Norms  for  each  Pass-Fail 
measure  were  established  on  a  student  population,  and  scores  of  the  bottom  5%  of  thal 
normative  population  on  each  measure  were  judged  to  be  unsatisfactory.  These  were 
used  to  establish  cutoff  scores  for  future  student  populations,  which  is  considered 
a  defensible  procedure  for  Pass-Fail  application . M  The  procedure  was  applied 
experimentally  on  the  first  eight  classes,  118  sti dents,  completing  the  new  FSS 
training  program.  Seven  students,  six  percent  of  the  group,  had  a  combination  of 
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THE  NEW  FSS  TRAINING  PROGRAM 
PERFORMANCE  STANDARDS  FOR  PASS-FAIL  DETERMINATIONS 
E.  W.  PICKREL,  PH.D. 

OFFICE  OF  AVIATION  MEDICINE 

INTRODUCTION 

This  report  describes  and  documents  Pass-Fail  procedures  for  the  new 
FSS  Training  Program.  It  describes  the  derivation  of  performance  standards  and 
presents  an  example  problem  to  demonstrate  their  application. 

The  report  has  the  following  major  sections 

-  Introduction 

-  Overview 

-  Performance  Standards 

-  Appendices 

-  Tables 

The  appendices  provide  rationale  and  technical  information  for  the  reader 
interested  in  details  of  the  new  program.  The  report  can  be  read  without 
reference  to  the  appendices. 

Background 

In  1977  a  Congressional  Committee,  the  House  Government  Affairs  Committee, 
released  a  report  recommending  that  the  FAA  improve  its  selection  and  training 
procedures  for  air  traffic  controller  specialists  and  reduce  the  $13.8  millions 
lost  annually  because  of  the  unacceptably  high  number  of  Air  Traffic  Control 
Specialists  who  drop  out  of  the  program  after  several  years  of  training. 


The  report  suggested  that  the  FAA  review  the  criteria  and  selection 
devices  used  by  the  Civil  Service  Commission  and  develop  a  test  battery  that 
will  more  accurately  reveal  whether  a  candidate  will  succeed  as  a  controller. 

The  committee  further  recommended  that  criteria  for  screening  and  eliminating 
unsuccessful  students  be  established  and  used  at  the  ATC  Academy  as  well  as 
later  in  the  training  program  to  ensure  that  potentially  unsuccessful  controllers 
are  eliminated  early  in  the  training  process. 

The  FAA  is  responding  to  these  recommendations.  As  an  initial  step, 
the  Office  of  Aviation  Medicine  has  created  new  selection  tests  for  the  Civil 
Service  Commission  to  use  after  a  person  first  applies  for  a  position.  These 
include  the  ATC  Occupational  Knowledge  Test  and  the  Multiplex  Controller 
Aptitude  Test.  These  tests  have  been  validated  against  such  criteria  as  ATC 
Academy  training,  supervisory  ratings  on  the  job,  career  progression,  and 
attrition.  The  available  data  show  that,  with  application  of  these  tests,  most 
of  the  applicants  entering  the  Flight  Service  Station  program  should  succeed 
as  air  traffic  controller  specialists.  The  failure  rate  at  the  Academy 
should  be  relatively  low  for  those  who  enter  through  Civil  Service 
Commission  competitive  testing  channels. 

The  House  Government  Affairs  Committee  report  had  further  stated  that 
the  present  training  system  appears  to  impede  the  elimination  of  unqualified 
students,  forcing  a  decision  to  terminate  the  individual  into  the  later  stages 
of  training,  and  recommended  that  criteria  for  screening  and  eliminating 
unsuccessful  students  be  established  and  used  at  the  ATC  Academy  as  well  as 
later  in  the  training  program  to  ensure  that  potentially  unsuccessful 
controllers  are  eliminated  early  in  the  training  process. 
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The  new  FSS  Training  Program  is  responsive  to  that  recommendation. 

Training  itself  is  a  feature  of  an  improved  screening  system  that  will  extend 
beyond  the  initial  selection  testing  by  incorporating  measures  of  the  candidates' 
performance  during  initial  training.  Parts  of  the  FSS  training  program  will  be 
included  in  the  applicant  screening  process. 

This  effort  was  requested  and  coordinated  by  the  Office  of  Personnel  and 
Training,  Headquarters  FAA.  Office  of  Aviation  Medicine  personnel,  consultants, 
and  members  of  the  ATC  Academy  Flight  Service  Station  Sections  *  Development  Unit 
cooperatively  developed  new  FSS  skills  tests,  paper  and  pencil  simulations  of 
laboratory  problems.  Personnel  at  operational  Flight  Service  Stations  across 
the  country  were  most  cooperative  in  taking  these  tests  for  validation,  relation 
to  on-the-job  performance,  and  standardization  purposes.  The  Chief  of  the  Flight 
Service  Station  Section  assigned  many  of  the  instructional  staff  to  the  Development 
Unit,  to  assist  in  such  tasks  as  designing  the  new  classroom  and  laboratory 
facilities,  developing  the  new  instructional  materials,  laboratory  problems  and 
procedures.  Section  meetings  were  held  to  keep  all  personnel  informed  regarding 
this  team  effort,  the  role  of  such  materials  as  the  new  measurement  devices  and 
procedures  being  developed  to  assure  heightened  validity,  reliability  and  general 
effectiveness  within  the  new  Pass-Fail  structure,  and  to  prepare  all  FSS  Section 
members  for  that  moment  when  the  new  training  program  would  become  operational. 

As  the  first  class  began,  the  FSS  Section  Chief's  observation  was  that  the  new 
students,  including  transfers  from  the  Terminal  and  Enroute  options,  seemed 
to  be  very  receptive  and  well  motivated  in  the  new  facility,  stimulated  to  perform 
well  in  the  new  program. 
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OVERVIEW 


General 

The  new  FSS  program  is  aimed  at  providing  previously  screened  candidates 
with  a  training  and  evaluation  curriculum  that  should  ensure  that  the  great 
majority  of  them  will  be  ready  for  journeyman  assignments,  yet  eliminate  the 
few  whose  performance  in  training  indicates  a  high  probability  of  failure  on- 
the-job.  As  such,  it  is  responsive  to  the  House  Government  Affairs  Committee 
recommendation. 

The  program  features  Pass-Fail  evaluation  keyed  to  the  phases  of  training. 
Phase  II  training  is  identified  as  the  initial  ATC  Academy  Pass-Fail  point, 
with  Pass-Fail  determinations  to  be  made  at  the  end  of  Phase  II  and  Phase  III. 
When  there  is  no  information  to  indicate  that  the  student  has  not  completed  a 
phase  of  training  in  a  satisfactory  manner,  the  student  will  be  advanced  to 
the  next  phase.  Scores  earned  in  Phase  II  will  be  used  to  determine  advance¬ 
ment  to  Phase  III,  but  not  be  used  as  part  of  the  composite  for  Phase  III. 

Phase  III  is  designed  to  qualify  the  developmental  to  perform  the  duties 
of  each  position  of  operation.  These  include  the  duties  of  Weather  Observer, 
Broadcast,  Teletype,  Flight  Data,  Preflight  and  Inflight  positions  of  operation 
plus  Emergency  Services  to  Aircraft.  The  latter  three  include  functions  which, 
if  not  fulfilled,  could  have  potentially  catastrophic  results,  and  impact  the 
safety  of  the  air  traffic  system.  These  also  are  the  more  complex  operational 
activities,  requiring  some  performance  of  most  of  the  duties  of  the  other 
positions,  and  thus  are  at  the  top  of  the  FSS  positions  hierarchy. 
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Phase  III  Evaluation  Measures 

The  new  Phase  III  student  evaluation  procedure  is  designed  to  be  both  an 
integral  part  of  the  Phase  III  training  and  a  component  of  the  screening  process. 
This  requires  that  it  be  composed  of  measurement  instruments  that  are  representa¬ 
tive  of  the  job  areas  found  in  Flight  Service  Stations  and  that  it  include  enough 
measures  to  permit  reliable  Pass-Fail  assessment.  Further,  it  must  assess  both 
job  knowledge  and  readiness  for  job  performance.  Thus,  it  must  include  measures 
of  academic  classroom  achievement  and  ability  to  perform  job-like  tasks  in  a 
laboratory  environment. 

Table  I  page  21,  lists  the  activities  and  measurement  instruments  included 
in  Phase  III  evaluation.  The  number  of  measures  available  to  assure  valid  and 
reliable  assessment  is  displayed  also. 

Academic  classroom  performance  is  measured  by  block  tests  and  achievement 
tests.  These  are  expert  prepared  multiple-choice  tests.  The  fact  that  average 
scores  for  current  classes  are  at  the  high  80  or  90  percent  level  indicates 
that  training  continues  until  high  student  performance  is  achieved.  Consequently 
all  students  completing  Phase  II  successfully  enter  Phase  III  with  a  similar 
academic  foundation  in  the  FSS  subject  matter  irregardless  of  considerations 
such  as  sex.  For  the  academic  area,  this  practice  conforms  with  the  House 
Government  Affairs  Committee  recommendations. 

Ability  to  perform  job  tasks  is  assessed  by  laboratory  problems  for  each 
of  the  job  activity  areas  and  by  FSS  skills  tests  for  the  critical  areas  of 
Flight  Bata,  Preflight,  Inflight  and  Emergency  Services.  A  student  will 
accomplish  7  block  tests,  10  achievement  tests,  28  graded  laboratory  problems 
and  4  FSS  skills  tests  for  a  total  of  49  independent  measures. 
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Procedural  Changes 

For  the  new  FSS  Training  Program  to  operate,  significant  changes  were 
required  in  facilities  and  procedures  for  the  laboratory  phase.  The  new 
training  program  provides  expanded  laboratory  space  and  an  increase  in  number 
of  graded  problems  for  each  of  the  critical  Preflight,  Inflight  and  Emergency 
Services  activities. 

A  different  instructor  scores  the  student  on  each  graded  laboratory 
problem,  and  without  awareness  of  other  instructors'  ratings  of  the  student's 
earlier  performances.  Use  of  multiple  raters,  each  one  independently  grading 
each  student,  adds  objectivity  to  the  grading  process.  It  is  fair  to  the 
student,  yet  protects  instructors  from  student  complaints  of  bias  directed 
against  them.  Additional  desirable  procedural  actions  are  presented  unde. 
Recommendations. 

Appendix  1,  page  9,  is  a  statement  about  laboratory  training  that 
expands  upon  changes  in  laboratory  procedures. 

FSS  Skills  Tests 

FSS  skills  tests  have  been  developed  to  help  support  the  evaluations  given 
in  the  laboratory.  These  are  paper-and-pencil  tests  that  simulate  the  laboratory 
problems.  The  scoring  is  accomplished  by  objective  keys,  a  procedure  that  serves 
to  minimize  the  impact  of  instructor  biases.  These  new  tests  have  been  administered 
to  a  considerable  sample  of  developmental  and  journeymen  at  operational  Flight 
Service  Station  facilities  and  to  a  comparable  size  group  of  FSS  students  at  the 
ATC  Academy  for  validation,  relation  to  on-the-job  performance,  and  for  standard¬ 
ization  purposes.  Since  norms  have  been  developed  to  describe  performance  of 
these  groups,  a  new  student's  scores  on  the  skills  tests  can  be  compared  to  those 
of  FSS  field  personnel  as  well  as  to  those  of  other  FSS  students  at  the  Academy. 
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Thus  Pass-Fail  evaluations  are  based  on  actual  job  performance  data. 

Unacceptable  student  performances  resulting  in  failure  scores  will  be  determined 
by  reference  to  the  normative  data. 

Appendix  2,  page  11,  contains  a  detailed  description  of  the  process  by  which 
the  normative  data  was  acquired  and  the  norms  established. 

PERFORMANCE  STANDARDS 

The  establishment  of  standards  for  applying  Pass-Fail  criteria  to  Phase 
III  student  performance  has  been  based  on  detailed  analysis  of  data  obtained 
on  253  active  FSS  air  traffic  control  specialists,  the  last  eight  ATC  Academy 
classes  (78-1  through  78-8)  in  the  old  training  program,  and  the  first  eight 
classes  (78-01  through  9006)  in  the  new  training  program.  The  technical  analysis 
that  supports  the  standards  and  procedures  described  herein  is  provided  in 
Appendix  3,  page  15.  The  obiective,  in  line  with  the  House  Government  Affairs 
Committee  recommendations,  has  been  to  provide  an  initial  rationale  and  procedure 
that  will  minimize  failures,  yet  eliminate  students  most  unlikely  to  succeed 
on  the  job.  A  basic  assumption  underlying  the  standards  is  that  the  ATC 
Occupational  Knowledge  Test  and  the  Multiplex  Controller  Aptitude  Test  will  be 
used  in  the  selection  of  candidates  for  the  FSS  air  traffic  controller  specialist 
posi tions . 

Briefly,  with  this  cutoff  procedure,  a  student  who  fails  both  the  laboratory 
problems  and  skills  test  in  the  Pilot  Briefing  or  Inflight  or  Emergency  Services 
positions  is  subject  to  failure.  These  include  functions  which,  if  not  fulfilled, 
could  have  potentially  catastrophic  results  and  impact  the  safety  of  the  air  traffic 
system.  A  student  who  fails  to  achieve  a  passing  score  on  the  final  Phase  Grade 
also  is  subject  to  failure.  This  Phase  composite  is  a  weighted  average  of  all 
phase  scores  including  academic  block  tests,  the  Pilot  Briefing,  Inflight,  and 
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Emergency  Services  skills  tests,  and  the  graded  laboratory  problems  for 
all  positions.  This  is  the  procedure  as  established  thus  far. 

This  procedure  is  recommended  to  identify  those  whose  performance  is 
unacceptable,  A  reviewer  then  should  determine  the  degree  of  consistency 
among  the  student's  scores,  to  assure  that  sub-par  performance  is  not  result¬ 
ing  from  chance  variations  in  measurement.  Since  a  student's  ATC  career  is 
at  stake  and  judgments  made  during  this  review  have  a  high  level  of  personal 
consequence,  a  recommendation  is  that  the  review  be  carried  out  by  the  FSS 
Section  Chief  or  his  designated  delegate.  Office  of  Aviation  Medicine  and 
FSS  Academy  personnel  will  continue  research  and  development  work  to  further 
refine  and  strengthen  the  cut-off  standards. 
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APPENDIX  I 


LABORATORY  PROCEDURES 

The  laboratory  is  an  attempt  to  create,  at  the  ATC  Academy,  something 
similar  to  the  operational  environment.  It  provides  the  opportunity  for  each 
student  to  practice  what  has  been  learned  in  the  classroom,  and  is  the  best 
available  phase  of  ATC  Academy  training  for  measurement  of  the  ability  to  per¬ 
form  the  duties  of  the  various  FSS  positions  of  operation.  Most  graded 
laboratory  problem  scores  are  derived  from  over- the-shoulder  observations, 
using  checklists  that  were  derived  from  analysis  of  the  tasks  involved 
during  operational  work  performance.  The  observer  indicates  whether  each 
step  has  been  successfully  achieved.  Through  checking  of  task  elements 
(steps),  these  checklists  minimize  subjectivity  in  the  assessment  process. 

Such  assessments  are  more  reliable  than  descriptive  or  numerical  rating  scales. 

Evaluation  forms  in  the  old  training  program  did  not  incorporate 
instructor's  comments  into  the  grading  process.  Instructor  comments  were 
available  on  the  back  of  the  forms,  but  were  not  reduced  to  numerical  scores. 
Instructor  evaluations  were  being  used  as  part  of  student  aggregate  scores 
in  the  terminal  and  enroute  ATC  Academy  courses,  adding  a  new  element  to  the 
aggregate  scores  and  increasing  the  stability  of  the  Pass-Fail  assessment  on 
each  problem.  An  incorporation  of  the  instructor's  numerical  assessment  of 
each  student's  performance  on  a  problem,  and  predicted  potential  performance 
on  future  problems,  was  added  to  the  checklists  to  be  used  in  the  new  FSS 
Training  Program. 

A  weakness  of  the  old  training  program's  laboratory  phase  was  the  short¬ 
age  of  work  space  for  positions,  which  resulted  in  a  student  receiving  only 
two  practice  problems  and  one  graded  problem  in  the  primary  positions. 
Laboratory  problems  are  individual  testing  situations  that  provide  opportunity 
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for  attention  to  much  detail,  but  the  process  is  time-consuming,  and  the  use 
of  single  problem  scores  permitted  too  much  opportunity  for  chance  elements 
to  enter  into  Pass-Fail  judgments.  The  new  FSS  Training  Program  provides 
for  expanded  laboratory  space  and  an  increase  in  the  number  of  graded  problems 
per  position.  For  example,  there  are  four  graded  problems  for  each  of  the 
primary  Preflight,  Inflight,  and  Emergency  Services  activities.  A  different 
instructor  scores  the  student  on  each  problem,  and  without  awareness  of 
other  instructors'  ratings  of  the  student's  earlier  performance.  A  procedure 
of  using  multiple  raters,  each  one  independently  grading  each  student,  adds 
objectivity  to  the  grading  process.  It  also  provides  an  excellent  defense 
against  the  student  who  receives  a  failure  in  the  course  and  sends  out  letters 
of  complaint,  saying  one  instructor  was  responsible  for  this  and  did  it  out 
of  spite  because  of  the  student's  race,  religion,  or  sex. 


APPENDIX  2 

EVALUATION  OF  THE  FSS  SKILLS  TESTS 

Information  concerning  the  suitability  and  the  operational  usefulness  of 
the  new  FSS  skills  tests  was  obtained  empirically  by  carefully  selecting  an 
Operational  Facility  sample  of  practicing  Air  Traffic  Controllers  and  a  sample, 
comparable  in  size,  of  ATC  Academy  students.  Demographic  data,  test  performance 
scores  and  intercorrelations  for  the  two  samples  are  presented  in  the  tables 
of  this  report. 

Operational  Facility  Sample 

Some  characteristics  of  the  operational  facility  sample  are  described  in 
Table  2,  page  22.  Almost  90%  were  males  and  over  30  years  of  age.  Approximately 
75%  were  of  grade  GS-11  or  higher.  Seventy-two  percent  had  attended  college, 
and  17%  of  this  sample  held  a  bachelors  or  masters  degree.  Eighty-one  percent 
had  36  months  or  more  experience  in  the  FSS  activity,  and  70%  were  full 
performance  level  personnel.  Forty-two  percent  held  some  kind  of  pilot  certifi¬ 
cate,  with  25%  holding  a  commercial  or  instructor's  rating.  Relationships 
between  skills  test  performances  and  various  kinds  of  experience  are  described 
in  Table  3-6,  page  23-36.  There  is  improvement  in  skills  test  mean  performance 
scores  with  increased  job  experience,  i.e.  between  performances  of  those  with 
less  than  two  years  of  FSS  experience  and  those  with  2  to  3  years  experience, 
but  generally  little  or  no  gain  after  that  time.  Personnel  in  that  larger 
group,  the  older  people,  include  several  kinds  of  administrators  and  staff 
personnel  whose  other  duties  can  cause  them  to  lose  touch  with  technical 
details  relating  to  day-to-day  servicing  of  aircraft.  Increases  in  GS-grade 


level  show  a  similar  relation  to  mean  skills  test  performance  scores.  Most 
of  the  improvement  in  performance  peaks  at  the  GS-11  level,  and  generally 
there  is  little  or  no  gain  for  grades  GS-12  and  up.  Those  holding  commerical 
and  instructor  pilot  certificates  perform  better  on  the  skills  tests  than 
those  who  are  less  experienced.  Those  with  Enroute  Flight  Advisory  Service 
(EFAS)  training  consistently  perform  somewhat  better  than  those  lacking  that 
additional  training.  Those  receiving  ATC  Academy  training  more  than  a  year 
ago  score  better  than  those  who  never  received  such  training,  but  the  relatively 
inexperienced  personnel  who  received  their  ATC  Academy  training  within  the  last 
year  did  not  perform  as  well  as  more  experienced  personnel  who  never  received 
such  training.  Those  in  facilities  having  a  full  time  Evaluation  Professional 
Development  Specialist  (EPDS)  scored  somewhat  higher  on  the  Preflight,  Inflight, 
and  Emergency  Service  Skills  Tests  than  those  at  facilities  without  the  services 
of  a  full  time  specialist.  Differences  in  skills  test  performances  between 
sexes  can  be  accounted  for  by  chance;  there  is  no  significant  difference  from 
this  sample  on  these  tests.  Intercorrelations  of  past  experience  and  performance 
on  the  skills  tests  are  presented  in  Table  7,  page  27  and  show  a  significant 
relation  between  skills  test  performance  and  success  in  the  FSS  option. 

ATC  Academy  Sample 

Some  characteristics  of  the  ATC  Academy  student  sample  are  described  in 
Table  8,  page  28.  Approximately  43  percent  were  over  thirty  years  of  age,  and 
207.  were  females.  Seventy-seven  percent  claimed  some  college  education,  and 
277.  claimed  a  bachelors  or  masters  degree.  Seventy-seven  percent  claimed  prior 
ATC  experience  and  427.  held  some  type  of  pilot  certificate. 


Table  9,  page  29,  presents  summary  statistics  and  intercorrelations  of 
performance  on  FSS  skills  tests  and  graded  laboratory  problems  for  the  student 
population  in  the  old  training  program.  The  FSS  skills  tests  correlate  well 
with  Fundamentals  of  Air  Traffic  Control,  a  multiple-choice  measure  of  general 
information  in  air  traffic  control,  as  well  as  with  the  laboratory  average 
grade  and  with  each  other.  The  Preflight  Briefing  Skills  Test  is  scored  Rights 
Only,  and  correlates  .384  with  the  lab  average  score.  This  test  presents  in 
written  form  the  kind  of  dialogue  that  takes  place  when  a  pilot  communicates 
by  radio  or  telephone  for  a  briefing.  The  student  is  supplied  with  weather 
data  sheets  for  which  to  supply  this  kind  of  information,  plus  multiple  choice 
questions  regarding  the  appropriate  responses  to  the  pilot's  questions.  The 
Inflight  Skills  Test  presents  in  written  form  the  kind  of  dialogue  that  takes 
place  between  those  working  this  position  and  pilots  who  are  airborne.  The 
student  is  provided  Weather  Data  Sheets  from  which  to  provide  this  type  of 
information  and  an  Action  List  of  nineteen  possible  actions  from  which  to 
select  responses  for  the  questions.  As  multiple  actions  should  be  taken  in 
most  situations,  the  student  may  erroneously  omit  some  actions  that  should  be 
identified,  and  include  actions  that  would  be  inappropriate  or  wrong.  These 
omission  and  commission  errors  seem  to  be  quite  independent  negative  scores 
or  error  measurement.  The  data  suggest  that  a  combining  of  the  two  scores  to 
form  a  new  "omit  plus  wrongs"  score  would  provide  an  excellent  measure  of  the 
student's  performance  on  the  Inflight  Skills  Test. 

Emergency  Services  Skills  Test  is  a  VOR  orientation  problem,  utilizing 
a  branching  technique  to  present  the  student  with  optional  paths  to  follow 
in  locating  a  lost  aircraft.  If  the  student  makes  a  poor  decision,  opportunities 
are  provided  in  the  form  of  Minor  Error  paths,  for  a  return  to  the  better 
"Major  Decision"  path.  Phraseology  questions  also  are  provided  in  the  test. 
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The  Phraseology  and  Major  Decisions  subscores  seem  to  be  parallel  measures 
of  the  same  skill.  A  combining  of  these  should  provide  a  more  reliable 
single  measurement.  The  Minor  Error  path  provides  much  logical  appeal  to 
specialists  in  air  traffic  control  and  is  needed  to  maintain  the  simulation. 
A  total  score  which  combines  these  sub-scores,  is  recommended  for  use  in 
determining  Pass-Fail  for  the  Emergency  Skills  Test. 


APPENDIX  3 


EVOLVING  PERFORMANCE  STANDARDS 

Scores  of  operational  personnel  and  those  of  past  students  r.ay  be  used  to 
create  standards  of  performance  that  new  students  must  meet  to  become  eligible 
for  acceptance  into  the  operational  facility  work  force.  For  example,  the 
total  work  force's  capabilities  will  gradually  improve  if  an  entrance  eligibil¬ 
ity  requirement  for  new  personnel  is  that  their  test  performance  must  be  better 
than  that  of  the  bottom  57.  of  the  current  work  force.  Students  nearing 
completion  of  ATC  Academy  training  might  then  be  required  to  meet  this  standard 
to  demonstrate  their  readiness  for  an  operational  assignment.  The  operational 
FSS  facility  sample,  Table  2,  and  the  ATC  Academy  FSS  student  sample,  Table  8, 
had  remarkably  similar  c:er.O_rs.phic  characteristics.  Performances  of  students 
in  the  first  eight  classes  78-01  through  9006,  in  the  new  training  program 
should  provide  a  stable  statistical  base  for  derivation  of  Pass-Fail  cutoffs. 

Laboratory  Grades 

The  development  of  improved  laboratory  procedures  for  the  new  training 
program  have  been  described  in  Appendix  1.  The  number  of  graded  problems  was 
increased  fourfold,  instructor  evaluations  were  incorporated  into  the  grading 
process,  and  stricter,  more  objective  quantitative  grading  procedures  were 
introduced  to  provide  an  improved  scoring  base  for  identifying  the  weak 
students.  Table  10-12  (pages  30-33  present  distributions  of  laboratory 
averages  in  Pre-Flight,  Inflight,  and  Emergency  Services  for  samples  of 
students  in  the  old  and  the  new  training  programs.  The  Preflight  Lab  Problem 
mean  dropped  from  95.72  in  the  old  program  to  80.12  in  the  new  program. 

The  distribution  of  scores  in  the  old  program  ranged  from  80  to  100,  but  in 
the  new  program  the  range  is  from  a  52  to  a  91,  which  will  be  more  useful  for 
identification  of  weak  students.  The  Inf light  Lab  Problem  mean  dropped  from 


94.40  to  an  84.36.  The  distribution  of  scores  in  the  old  program  ranged  from 


65  to  100  and  with  only  three  students  scoring  below  84.  In  the  new  program 
it  ranges  from  69  thru  98,  with  a  half  dozen  students  scoring  below  72.  The 
Emergency  Services  Lab  Problem  scores  were  a  binary  pass-fail  in  the  old 
training  program,  and  scores  in  the  new  program  range  from  67  through  98 
with  a  mean  of  87,69.  The  cutoff  scores  for  each  distribution  in  the  new 
training  program  below  which  approximately  five  percent  of  the  students  scored 
is  given  in  Table  18,  page  39.  The  cutoffs  for  passing  Preflight,  Inflight, 
and  Emergency  Services  are  laboratory  problem  score  averages  (four  problems) 
of  70,  72,  and  76,  respectively,  representing  6.8%,  5.1«,  and  5.8%  of  the 
students  from  these  classes. 

Skills  Tests 

Evaluation  of  the  FSS  skills  tests  has  been  presented  in  Appendix  2. 

The  distributions  of  the  Preflight,  Inflight,  and  Emergency  Services  Skills 
Test  scores  for  the  Operational  Facility,  old  training  course,  and  new  training 
course  samples  are  presented  in  Tables  13-17,  (pages  33-38  ) .  As  shown  in 
Table  13,  page  33  ,  performance  of  students  in  the  new  training  program  on  the 
Preflight  Briefing  Skills  Test  was  slightly  better  than  that  of  students  in 
the  old  training  program,  and  persons  from  operational  facilities  scored  some¬ 
what  higher  than  either  student  group.  Tables  15-17  present  distributions  of 
totals  scores  on  the  Inflight  and  Emergency  Services  skills  tests.  The 
Inflight  totals  score  is  negative,  a  sum  of  wrongs  and  omissions.  The  Emergency 
Services  total  score  is  a  weighted  composite,  with  major  decisions  assigned  a 
weight  of  5,  minor  errors  a  weight  of  3,  and  phraseology  a  weight  of  1.  The 
improved  performance  of  the  students  in  the  new  training  program  is  striking. 
Those  in  the  new  training  program  averaged  17  fewer  mistakes  on  the  Inflight 
skills  test,  and  averaged  ten  points  better  on  the  Emergency  Services  test 
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than  those  in  the  oLd  training  program 


Table  18  contains  the  cutoff  score  for  each  Skills  Test  for  classes 
78-01  to  9006.  These  cutoffs  for  Preflight,  Inflight,  and  Emergency  Services 
are  11,  32,  and  15,  respectively,  representing  10.27.,  6.87.,  and  5.97.  of  the 
students  in  these  classes. 

Since  the  Skills  Tests  are  to  be  incorporated  into  the  calculation  of 
the  final  phase  grade  for  future  classes,  it  was  desired  to  transform  the 
raw  score  for  these  tests  to  a  0-100  scale  where  the  cutoff  score  would  be 
given  a  value  of  70.  Table  19  gives  the  converted  grades  for  each  of  the 
Skills  Tests.  The  raw  scores  above  the  cutoff  score  were  scaled  in  equal 
intervals  from  70  to  100,  while  the  scores  below  the  cutoff  score  were 
scaled  in  equal  intervals  to  0. 

Phase  Grades 

The  final  phase  grade  is  derived  from  a  weighted  composite  of  all  phase 
grades,  with  57.  allocated  to  academic  block  tests,  157.  to  secondary  position 
graded  laboratory  problems,  407.  allocated  to  primary  position  graded  laboratory 
problems  and  407.  allocated  to  primary  position  skills  tests.  The  sum  of  these 
scores,  a  weighted  composite,  is  then  converted  to  a  phase  graae.  ine  weighted 
composites  are  calculated  according  to  the  following  linear  combination: 

57.  Academic  Average  (average  of  all  block  tests); 

157.  Graded  laboratory  problem  averages  for  Weather  Observer,  Teletype, 
Broadcast,  and  Flight  Data; 

157.  Preflight  Laboratory  Average  (four  graded  problems); 

157.  Preflight  Skills  Test  Converted  Score  (Table  19); 

12.57.  Inflight  Laboratory  Average  (four  graded  problems); 

12.57.  Inflight  Skills  Test  Converted  Score  (Table  19); 

12.57.  Emergency  Services  Laboratory  Average  (four  problems);  and 

12.57.  Emergency  Services  Skills  Test  Converted  Score  (Table  19). 


I 


Each  of  the  above  scales  is  based  on  a  grading  metric  from  0  to  100. 

In  order  to  provide  some  normative  data  for  future  classes,  all 

weighteu  composite  scores  were  computed  for  classes  7>i-ol  to  4006 

according  to  the  above  linear  combination.  These  were  standardized  using 

the  mean  (32.84)  and  standard  deviation  (4.54)  of  the  weighted  composite 

scores  (see  Table  20).  Then  they  were  re-scaled  for  development  of  the 

new  table  for  converting  weighted  composite  scores  to  final  Phase  Grade 

(Table  21).  The  ..ojireu  transformation  is: 

Y  =5.44  (X-82.S4)  +  79. -+4, 

!  4.54 

where  X  is  the  weighted  composite  score  and  Y  is  the  final  phase  grade. 

This  transformation  i  lentil'll.  =  5. 17.  of  the  students  from  the  classes  78-01 
to  9006  as  scoring  below  70,  and  thus  labeled  as  failure  ..  The  scaled 
scores  in  Table  21  are  intended  to  be  used  with  future  Academy  classes. 

Failure  Criteria 

Two  ways  to  fail  Phase  1  are  a  Position  failure  and  a  Phase  Grade  failure. 
Position  failure:  A  student  who  fails  both  the  skills  test  and  the  laboratory 
problems  in  the  Preflight  position  or  Inflight  position  or  Emergency  Services 
position  fails  the  phase.  The  Phase  Grade  entry  for  that  student  will  be  the 
word  FAIL,  with  no  numerical  score. 

Phase  Grade  failure:  A  student  with  a  final  Phase  Grade  below  70  fails  the 
course.  That  final  grade  is  derived  from  a  weighted  composite  of  all  phase 
grades . 

These  are  listed  in  Table  18. 

Table  22  lists  all  the  students  from  classes  78-01  to  9006  who  failed 
at  least  one  of  the  primary  position  Skills  Tests,  or  Laboratory  problem 
averages  or  Phase  Grade.  In  all,  38  students  or  32.27.  of  the  students  failed 
at  least  one  of  these.  Of  these  38  students,  six  students  were  Phase  Grade 
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failures.  One  of  these  students  (9006-5)  also  was  a  position  failure  in 
Preflight.  One  student  (9004-10)  was  also  identified  as  a  Preflight  Position 
failure.  Thus,  a  total  of  seven  students  (5.9%)  in  classes  78-01  to  900b 
would  have  failed  the  program  using  the  above  criteria. 

Summary  of  Data,  All  Variables 

Table  23,  page  44  ,  presents  summary  statistics  and  intercorrelations  of 
performance  on  block  tests,  graded  laboratory  problems,  and  the  FSS  skills 
tests  for  the  student  population  in  the  new  training  program.  Scores  on  the 
graded  laboratory  problems  for  the  several  positions  correlate  well  with  each 
other,  and  generally  higher  than  the  laboratory  problems  in  the  old  training 
program.  The  measures  of  student  performance  in  the  training  program  correlate 
well  with  the  final  Phase  Grade.  This  Phase  Grade  is  the  best  measure  present 
here  for  use  in  pass-fail  determinations.  The  classroom  block  test,  laboratory 
and  skills  test  per; ormuncej  a  roc:  t .  correlate  t  airiy  well  witn  eacr.  ccr.er 
to  support  their  use  as  a  composite,  but  not  as  well  as  the  final  Phase  Grade. 

The  Multiplex  Controller  Aptitude  Test,  MCAT,  a  newl>  created  measure  for 
use  in  initial  screening  of  applicants,  correlates  ,49  with  the  Phase  Grade. 
This  test  was  designed  to  measure  aptitude  for  the  type  of  work  being  taught 
in  the  new  training  course,  and  a  correlation  of  this  magnitude  is  a  mutually 
supportive  indicator.  The  final  Phase  Grade  is  demonstrating  a  desirably  high 
relation  to  a  student's  aptitude  for  this  work.  Also,  those  who  counsel 
stumbling  students  during  training  might  use  their  MCAT  scores  to  see  if  the 
source  of  difficulty  lies  in  lack  of  aptitude  for  the  work. 

Comment 

Instructors  generally  try  to  provide  a  fair  and  just  evaluation  of  a 
student's  performance  during  the  grading  of  laboratory  problems.  Fair  grading 
practices  are  expected  to  continue  after  the  introduction  of  Pass-Fail  in  the 
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new  training  program.  The  new  training  program  introduces  new  graded  problems, 
but  the  same  instructors  must  still  compare  each  student’s  performances  against 
a  standard  of  expected  performances  for  evaluation  purposes.  There  is  no 
reason  to  expect  any  dramatic  upsurge  in  the  number  of  students  receiving 
failing  grades,  or  to  anticipate  a  failure  rate  greatly  different  from  that 
described  in  the  example  problem  presented  here.  A  good  instructor  should  be 
able  to  get  the  subject  matter  across  to  most  students.  There  may  be  a  few 
exceptions  within  specific  classes,  as  some  may  have  a  greater-than-normal 
number  of  weak  students  and  have  a  higher  failure  rate.  The  background 
information  on  the  students  in  the  example  problem  shows  that  they  were  a 
high  quality  group. 

As  the  training  course  is  changed  and  new  laboratory  problems  introduced, 
instructors  and  evaluators  go  through  a  period  of  learning  and  adjustment  with 
the  new  materials.  Since  t.ie  FSS  Skills  Tests  have  not  been  changed  and  their 
norms  are  based  on  a  group  from  operational  facilities,  it  is  recommended 
that  these  tests  carry  a  weight  equal  to  that  of  the  graded  laboratory  problems 
for  making  Pass-Fail  judgements  during  this  adjustment  period.  The  laboratory 
problem  5%  cutoff  per  position  may  be  obtained  by  first  averaging  each  student's 
graded  laboratory  problem  results  per  position,  then  ranking  all  the  available 
scores  and  identifying  the  bottom  5%  cutoff.  After  this  is  done,  following  the 
procedure  outlined  in  the  above  example  problem  will  identify  those  students 
whose  performance  is  unacceptable.  The  5%  cutoff  for  averaged  laboratory 
grades  will  be  based  on  a  very  small  sample  at  the  start,  and  should  be  re¬ 
computed  as  the  available  student  sample  increases  with  each  new  class,  so  that 
a  more  stable  cutoff  may  be  realized. 


TABLE  1 

MEASUREMENTS  AVAILABLE  IN  PHASE  III 
FOR 

PASS-FAIL  EVALUATIONS 


Activity 

Block  Tests 

Achievement 

Tests 

Graded  Laboratory 
Problems 

FSS 

Skills  Tests 

Broadcast 

1 

6 

Teletype 

1 

- 

Weather  Observer 

1 

3 

- 

Flight  Data 

1 

2 

1 

Preflight 

1 

3 

- 

1 

Inflight 

1 

*4 

1 

Emergency  Services 

1 

4 

1 
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TABLE  2 


POPULATION  CHARACTERISTICS 
FIELD  SAMPLE,  OPERATIONAL  FACILITIES 

Testing  Dates 
July-August  197? 


FAA  FLIGHT  SERVICE  STATIONS 


253  Persons 


AGE  IN  YEARS _ 

Interval  Number  of  Persons 


_ SEX _ 

Type  Number  of  Persons* 


36  A  over 

164 

31-35 

47 

26-30 

25 

20-25 

5 

No  Answe; 

12 

Male  221 
Female  27 
No  Answer  5 


Education  (Highest  Attained) 

Current  CS  Grade 

Level  Number  of  Persons 

Leve  1 

Number  of  Persons 

Masters  Degree 

3 

12  A 

up  20 

Bachelor  Degree 

•40 

11 

169 

College  3-u  years 

•4? 

10 

40 

2  vears 

40 

9 

K 

1  year 

43 

5-7 

1C 

High  School  Diploma 

69 

No  Answer  6 

No  Diploma 

3 

ATCS  EXPERIENCE 

( Months ) 

Option 

None 

1-11 

12-23 

24-25 

36  A  up  No  Answers 

FAA  FSS  FPL 

20 

9 

21 

24 

176  3 

GS-2152  FSS  Option 

5 

s 

11 

23 

205  1 

Enroute  Option 

12S 

17 

40 

32 

30  6 

Terminal  Option 

207 

19 

12 

5 

6  4 

Before  joining  FAA 

145 

15 

5 

14 

73  1 

Pilot  Certificate 

FAA 

ATC  Academy  Training 

When 

Number  of  Persons 

Tvpe  Number 

of  Persons 

Ins  true  tor 

18 

Graduated  1976-77 

31 

Commercial 

44 

1975  or  before 

93 

Priva  te 

27 

Did  not  attend 

124 

Student 

16 

No  Answer 

5 

None 

145 

No  Answer 

3 

EFAS  Training _ 

Number  of  Persons 


Yes  45 
No  206 
No  Answer  2 
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TABLE  3 


TEST  PERFORMANCES 

FIELD  SAMPLE,  OPERATIONAL  FACILITIES 
Testing  Dates  FAA  FLIGHT  SERVICE  STATIONS 


July-August  1977 

Tes  ts 


ATC  Fundamentals-  Rights 

100  items 

Preflight  Briefing-  Rights 
2  5  i  terns 

Inflight-Negative  Scores 

Wrongs-  172  items 
Omissions-  60  items 
Wrongs  +  Omissions 

Emergency  Skills  II-  Rights  weighted 

Major  Decisions-  10  items 
Minor  Decisions-  7  items 
Phraseology-  12  items 
TOTAL 


253  Persons 

N  Mean  Standard  Deviation 


218 

74.03 

9.96 

24  3 

15.22 

2.92 

229 

18.59 

8.00 

229 

23,08 

7.48 

229 

41,56 

11.93 

253 

12.55 

7.65 

253 

.24 

.81 

253 

4.64 

2.54 

253 

17.43 

9.97 

23- 


TABLE  4 


RELATING  TEST  PERFORMANCE 
TO 

POPULATION  CHARACTERISTICS 
FIELD  SAMPLE,  OPERATIONAL  FACILITIES 

Test  Performance  Mean  Scores 
versus 

Testing  Dates 
July-August  1977 


253  Persons 


Tests 

Months 

Experience , 

FSS  Option 

ATC  Fundamentals- 

Rights 

0-23 

N=  25 

14-35 

N=  23 

36  A  up 

N=  205 

100  items 

71.84 

79.17 

77.34 

Preflight  Briefing-  Rights 

25  items 

14.16 

15.26 

15.85 

Inf light- Nega  tive 

scores 

Wrongs 

20.31 

18.70 

18.31 

Omissions 

31.79 

26.65 

26.05 

Wrongs  +  Omissions 

47.69 

40.20 

41.09 

Emergency  Skills 

II-  Rights  weighted 
Major  Decisions 

11.76 

13.10 

13.13 

Phraseology 

4.83 

5.14 

5.03 

Total 

16.28 

18.38 

19.13 

Tes  ts 

Pilot  Experience  Levels 

None 

Student 

Private 

Commercial 

Instructor 

ATC  Fundamentals-  Rights 

N=  145 
75.28 

N=  16 
76.69 

N=  27 
75.37 

N=»  44 

32.30 

N=  18 

82.11 

Preflight  Briefing-  Rights 

15.57 

14.50 

15.07 

16.11 

17.11 

Inflight-  Negative  Scores 

Wrongs  plus  Omissions 

42.33 

42.87 

.43.04 

39.93 

35.13 

Emergency  Skills  II-  Rights  weighted 
Total 

16.79 

16.25 

15.81 

23.13 

20.17 

TABLE  5 


RELATING  TEST  PERFORMANCE 
TO 

POPULATION  CHARACTERISITCS 
FIELD  SAMPLE,  OPERATIONAL  FACILITIES 


Testing  Dates 

July-August  1977 

FAA  FLIGHT 

SERVICE 

STATIONS 

253  Persons 

Tests 

Present  GS  Grade  Levels 

ATC  Fundamentals-  Rights 

5  -  7 

N=  10 
74.60 

9 

N=  8 
82.88 

10 

N=  40 
75.8 

11 

N=  169 

84.5 

12  A  up 

N=  20 

75.35 

Preflight  Briefing-  Rights 

14.7 

14.63 

14.43 

15.97 

16.60 

Inflight-Negative  Scores 

Wrongs  +  Omissions 

52.0 

52.8 

42.13 

40.91 

41.33 

Emergency  Skills  II-  Total 
Rights  Weighted 

17.5 

19.86 

18.95 

19.29 

16.41 

Tests 

HAD 

EFAS  Training 

No 

Yes 

ATC  Fundamentals-  Rights 

N=  206 
76.76 

N=  45 

78.53 

Preflight  Briefing-  Rights 

15.60 

15.84 

Inflight-Negative  Scores 

Wrongs  +  Omissions 

41.69 

40.54 

Emergency  Skills  II-  Total 
Rights  Weighted 

18.49 

20.88 

-25- 
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TABLE  6 


RELATING  TEST  PERFORMANCE 
TO 

POPULATION  CHARACTERISTICS 
FIELD  SAMPLE,  OPERATIONAL  FACILITIES 


Testing  Dates 

July-August  1977  253  Persons 


Received  Academy  Training 


Tests 

Within  last  year 

Never 

More  than  year  ago 

ATC  Fundamentals-Rights 

N=  31 

73.77 

N=  93 
75.80 

N=  124 

79.06 

Preflight  Briefing-Rights 

14.19 

15.88 

15.90 

Inflight-Negative  Scores 

Wrongs  +  Omissions 

45.93 

42.25 

39.39 

Emergency  Skills  II-  Total 

Rights  Weighted 

17.27 

17.67 

20.41 

Tes  ts 

Does  facility  have 

full  time  EPDS ? 

No 

Yes 

N=  93 

N=  156 

ATC  Fundamentals-Rights 

77.26 

77.05 

Preflight  Briefing-Rights 

15.24 

15.91 

Inflight-Negative  Scores 

Wrongs  +  Onissions 

42.37 

41.10 

Emergency  Skills  II-  Total 

Rights  Weighted 

18.46 

19.17 

Tests 

Sex  of  Specialist 

Male 

Female 

N=  221 

N=  27 

ATC  Fundamentals-Rights 

77.43 

76.19 

Preflight  Briefing-Rights 

15.62 

16.30 

Inflight-Negative  Scores 

Wrongs  +  Omissions 

41.61 

41.90 

Emergency  Skills  II-  Total 

Rights  Weighted 

19.26 

18.00 

-26- 


INTERCORRELATION  OF  PAST  EXPERIENCE  AND  PERFORMANCE  ON  FSS  SKILLS  TESTS 
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TABLE  8 

POPULATION  CHARACTERISTICS 
ATC  ACADEMY  SAMPLE 


Testing  Dates 

September  1977-February  1978 


273  STUDENTS 


_ Age  In  Years 

Interval 

36  &  over 
31  -  35 
26  -  30 
20  -  25 
No  answer 


of  Persons 


TyPe 

Male 
Female 
No  answer 


No.  of  Persons 


Education  (Highest  Attained) _ 

No.  of  Persons 


Masters  Degree 
Bachelor's  Degree 
College  3-4  years 
2  years 
1  year 

High  School  Diploma 
No  diploma 
No  answer 


ATC  Experiences  (Months) 
None  1-11  12-23 


36  &  up 


No  answer 


FAA  GS  2152  FSS  FPL 
FSS  Options 
Enroute  Option 
Terminal  Option 
Before  joining  FAA 


_ Pilot  Certificate 

Type  No,  of  Persons 

Instructor  34 

Commercial  35 

Private  30 

Student  14 

None 


28 
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TABLE  10 


PREFLICHT  LAB  PROBLEM  AVERAGES 
ATC  ACADEMY  STUDENT  SAMPLES 
OLD  TRAININC  COURSE,  CLASSES  78-1  thru  78-8 
NEW  TRAINING  COURSE,  CLASSES  78-01  THRU  9006 


Training  Course 


Training  Course 


m 


TABLE  11 


INFLIGHT  LAB  PROBLEM  AVERAGES 
ATC  ACADEMY  STUDENT  SAMPLES 
OLD  TRAINING  COURSE,  CLASSES  '8-1  THRU  78-8 
NEW  TRAINING  COURSE,  CLASSES  78-01  THRU  9006 


Training 

Course 

Training 

Course 

Old 

New 

Old 

New 

100 

10 

80 

8 

99 

7 

79 

4 

98 

17 

1 

78 

2 

97 

17 

77 

3 

96 

12 

2 

76 

1 

2 

95 

14 

75 

1 

94 

14 

1 

74 

3 

93 

13 

1 

73 

2 

92 

5 

2 

72 

1 

1 

91 

5 

8 

71 

3 

90 

5 

7 

70 

2 

89 

5 

8 

69 

1 

88 

6 

7 

68 

87 

2 

5 

67 

86 

4 

7 

66 

85 

2 

12 

65 

1 

84 

1 

11 

64 

83 

3 

63 

82 

10 

62 

81 

1 

61 

N 

142 

118 

Mean 

94.40 

84.36 

SD 

5.16 

6.12 

31- 


TABLE  12 


EMERGENCY  SERVICES  LAB  PROBLEM  AVERAGES 
ATC  ACADEMY  STUDENT  SAMPLES 
OLD  TRAINING  COURSE,  CLASSES  78-01  THRU  9006 


Training  Course 

Training  Course 

Old 

New 

Old 

New 

100 

83 

3 

99 

82 

2 

98 

1 

81 

5 

97 

3 

80 

2 

96 

4 

79 

95 

5 

78 

1 

94 

5 

77 

4 

93 

6 

76 

l 

92 

7 

75 

2 

91 

10 

74 

l 

90 

6 

73 

89 

11 

72 

88 

5 

71 

2 

87 

9 

70 

1 

86 

9 

69 

85 

6 

68 

84 

6 

67 

1 

Fail 

(2.0) 

7 

Pass 

(1.0) 

135 

N 

142 

118 

Mean  1 , 

.049 

87.69 

SD 

.217 

6.30 

-32- 
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TABLE  13 


DISTRIBUTIONS  OR  TEST  SCORES 
PREFLIGHT  BRIEFING  SKILLS  TEST 
OPERATIONAL  FACILITY  AND  STUDENT  SAMPLES 
OLD  TRAINING  COURSES  SAMPLES  78-01  THRU  78-8 
NEW  TRAINING  COURSE  SAMPLES  78-01  THRU  9006 


PREFLIGHT 

OPERATIONAL 

STUDENT 

SAMPLES 

TEST 

FACILITY 

TRAINING 

COURSE 

SCORES 

SAMPLE 

Old 

New 

22 

2 

2 

21 

3 

1 

1 

20 

6 

1 

2 

19 

20 

2 

6 

18 

24 

7 

10 

17 

30 

15 

13 

16 

34 

29 

12 

15 

35 

41 

18 

14 

27 

32 

16 

13 

17 

38 

8 

12 

17 

35 

10 

11 

12 

19 

7 

10 

7 

16 

7 

9 

6 

12 

2 

8 

1 

14 

1 

7 

1 

2 

1 

6 

1 

7 

1 

5 

1 

4 

3 

2 

1 

0 

No  Response 

10 

1 

Total  N 

253 

273 

118 

Mean 

15.22 

13.055 

14.64 

SD 

9.97 

2.934 

3.08 

33 


TABLE  14 

DISTKIBUTION  OF  TEST  SCORES 
INFLIGHT  SKILLS  TEST 
OPERATIONAL  FACILITY  AND  STUDENT  SAMPLES 
ATC  ACADEMY  CLASSES  78-1  THRU  78-8 


Operational  Facilit 
Wrongs  (Omission 


Sample 


ATC  Academy  Students 


Omissions 

Wrongs+Omissions 

Wrongs 

Omissions 

1 

1 

1 

1 

3 

1 

1 

5 

4 

2 

1 

1 

5 

2 

4 

6 

4 

4 

5 

11 

1 

5 

1 

7 

6 

1 

6 

1 

1 

8 

2 

1 

7 

1 

7 

4 

9 

1 

1 

10 

4 

14 

1 

2 

6 

3 

5 

4 

9 

1 

1 

1 

14 

1 

1 

9 

15 

1 

2 

7 

1 

5 

2 

9 

1 

2 

8 

8 

4 

5 

6 

3 

1 

7 

11 

7 

3 

7 

3 

2 

4 

9 

11 

2 

5 

7 

10 

3 

5 

11 

34 

Wrongs+ 

Omissions 

1 


TABLE  14  CONTINUED 


DISTRIBUTION  Oh  TEST  SCORES 
INFLIGHT  SKILLS  TEST 
OPERATIONAL  FACILITY  AND  STUDENT  SAMPLES 
ATC  ACADEMY  CLASSES  78-1  THRU  78-8 


Operational  Facility  Samples _  ATC  Academy  Students 


Inf  1 igh  t 

Wrongs 

Omissions 

Wr ong s+Om i s s i ons 

Wrongs 

Omissions 

Wrongs+ 

Scores 

Omissions 

25 

7 

10 

1 

7 

7 

4 

24 

7 

8 

6 

12 

3 

23 

12 

10 

9 

17 

1 

22 

8 

11 

2 

10 

14 

2 

21 

13 

13 

5 

23 

3 

20 

6 

15 

9 

16 

19 

9 

20 

11 

26 

18 

12 

6 

10 

20 

17 

14 

11 

15 

18 

16 

11 

8 

20 

18 

15 

18 

11 

15 

10 

14 

15 

11 

18 

10 

13 

12 

2 

18 

5 

12 

13 

3 

17 

6 

11 

12 

5 

16 

4 

10 

10 

1 

21 

9 

6 

3 

7 

8 

1 

O 

7 

3 

1 

7 

6 

3 

3 

5 

2 

3 

4 

1 

3 

1 

2 

1 

1 

0 

No 

Response 

24 

24 

24 

1 

1 

1 

Total  N 

253 

253 

253 

273 

273 

273 

Mean 

18.59 

23.08 

41.576 

16.978 

21.746 

38.794 

SD 

8.00 

7.48 

11.926 

7.917 

6.401 

9.753 

1 

-35- 

TABLE  IS 


DISTRIBUTION  OF  TEST  SCORES 

INFLIGHT  SKILLS  TEST-WRONGS+  OMISSIONS 
OPERATIONAL  FACILITY  AND  STUDENT  SAMPLES 


TABLE  16 


DISTRIBUTION  OF  TEST  SCORES  EMERGENCY  SERVICES 
SKILLS  TEST 

OPERATIONAL  FACILITY  AND  STUDENT  SAMPLES 


Operational  Facility  Sample _  ATC  Academy  Students 


Eme  rgency 

Total 

Ma  j  o  r 

Phraseo- 

Minor 

Totals 

Major 

Phrase 

Services 

Decision 

Error 

Decisions 

logy 

Scores 

42 

11 

41 

1 

C 

40 

1 

13 

39 

2 

2 

36 

1 

1 

37 

2 

2 

36 

2 

4 

35 

2 

8 

34 

1 

2 

33 

6 

2 

32 

2 

1 

31 

8 

- 

30 

5 

9 

8 

31 

29 

4 

3 

28 

6 

31 

27 

10 

7 

26 

8 

4 

25 

6 

18 

4 

20 

24 

3 

1 

23 

5 

9 

22 

5 

15 

21 

13 

10 

20 

18 

39 

6 

53 

19 

11 

3 

18 

5 

- 

17 

5 

7 

16 

6 

24 

15 

18 

63 

13 

45 

14 

8 

3 

13 

14 

1 

12 

8 

1 

1 

13 

11 

2 

9 

9 

10 

3 

52 

7 

11 

48 

32 

9 

5 

7 

12 

9 

8 

4 

20 

6 

43 

7 

16 

20 

3 

33 

6 

8 

40 

3 

50 

5 

1 

42 

46 

11 

47 

40 

4 

1 

28 

7 

22 

3 

5 

27 

20 

2 

9 

2 

3 

29 

4 

7 

1 

8 

2 

4 

0 

19 

30 

20 

233 

2 

29 

2 

No 

Response 

0 

Total  N 

253 

273 

Mi  nor 
Error 


9 

264 


21.139  14.212  6.828 

_37 11.404  9.106  2.570 


.099 

.537 


Mean  17.43  12.55 

SD  9.97  7.85 


4.64 

2.54 


24 

81 


TABLE  17 


DISTRIBUTION  OF  TEST  SCORES  EMERGENCY  SERVICES 
SKILLS  TEST 

OPERATIONAL  FACILITY  AND  STUDENT  SAMPLES 


Operational 

Training 

Program 

Operational 

Training  Program 

Facility 

Old 

New 

Facility 

Old 

New 

42 

11 

18 

20 

18 

6 

41 

1 

5 

3 

19 

11 

3 

40 

1 

13 

8 

18 

5 

1 

39 

2 

2 

2 

17 

5 

7 

38 

1 

1 

1 

16 

6 

24 

37 

2 

2 

4 

15 

18 

13 

36 

2 

4 

6 

14 

8 

3 

1 

35 

2 

8 

12 

13 

14 

1 

1 

34 

1 

2 

4 

12 

8 

1 

1 

33 

6 

2 

6 

11 

2 

9 

32 

2 

1 

10 

3 

11 

31 

8 

2 

9 

5 

12 

30 

5 

8 

6 

8 

4 

6 

29 

4 

3 

3 

7 

16 

3 

2 

28 

6 

31 

15 

6 

8 

3 

1 

27 

10 

7 

4 

5 

1 

11 

1 

26 

8 

4 

4 

1 

7 

25 

6 

4 

1 

3 

5 

2 

24 

3 

1 

2 

2 

3 

4 

23 

5 

9 

3 

1 

2 

22 

5 

15 

5 

0 

19 

2 

21 

13 

10 

5 

N 

253 

273 

118 

Mean 

17.43 

21.39 

31.86 

SD 

9.97 

11.404 

9.28 

38 


TABLE  18 


CUTOFF  SCORES  FOR  SKILLS  TEST  AND 
LABORATORY  GRADES  FOR  ATC  ACADEMY  STUDENTS  FROM 
CLASSES  78-01,  78-02, 

9001  to  900b 


N  «  118 


Skills  Test 

Type  of  Scores 

Cutoff 

Students 

Below  Cutoff 

Number 

Percent 

Pre flight 

Rights 

Less  than  11 

12 

10.2 

Inflight 

Negative i  Wrongs 
plus  omissions 

Greater  than 
32 

8 

6.8 

Bnergency 

Services 

Rights 

Less  than  15 

7 

5.9 

Laboratory 

Pre flight 

Average 

Less  than  70 

8 

6.8 

Inflight 

Average 

Less  than  72 

6 

5.1 

Bnergency 

Services 

Average 

Less  than  76 

7 

5.8 

39 
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TABLE  20 


DISTRIBUTION  OF  PHASE  GRADES  FOR  CLASSES  78-01  to  9006 


Weighted  Composite  Scores  Frequency 


94.01 

- 

95 

0 

93.01 

- 

94 

1 

92.01 

- 

93 

0 

91.01 

- 

92 

3 

90.01 

- 

91 

0 

89.01 

- 

90 

6 

88.01 

- 

89 

4 

87.01 

- 

88 

8 

86.01 

- 

87 

7 

85.01 

- 

86 

13 

84.01 

- 

85 

9 

83.01 

- 

84 

9 

82.01 

- 

83 

11 

81.01 

- 

82 

9 

80.01 

- 

81 

8 

79.01 

- 

80 

6 

78.01 

- 

79 

5 

77.01 

- 

78 

8 

76.01 

- 

77 

1 

75.01 

- 

76 

4 

74.01 

- 

75 

2 

73.01 

- 

74 

2 

72.01 

- 

73 

0 

71.01 

- 

72 

2 

70.01 

. 

71 

0 

41 


TABLE  21 


TABLE  FOR  CONVERTING  WEIGHTED  COMPOSITE  SCORES 
TO 

PHASE  GRADES 

Weighted  Composite  Phase  Grade  Weighted  Composite 


Score 

Score 

100 

100 

99.17 

- 

99.99 

99 

81.67 

- 

82.49 

98.33 

- 

99.16 

98 

80.83 

- 

81.66 

97.50 

- 

98.32 

97 

80.00 

- 

80.82 

96.67 

- 

97.49 

96 

79.17 

- 

79.99 

95.83 

- 

96.66 

95 

78.33 

- 

79.16 

95.00 

- 

95.82 

94 

77.50 

- 

78.32 

94.17 

- 

94.99 

93 

76.67 

- 

77.49 

93.33 

- 

94.16 

92 

75.83 

- 

76.66 

92.50 

- 

93.32 

91 

75.00 

- 

75.82 

91.67 

- 

92.49 

90 

74.17 

- 

74.99 

90.83 

91.66 

89 

73.33 

- 

74.16 

90.00 

90.82 

88 

72.50 

- 

73.32 

89.17 

89.99 

87 

71.67 

- 

72.49 

88.33 

89.16 

86 

70.83 

- 

71.66 

87.50 

88.32 

85 

70.00 

- 

70.82 

86.67 

87.49 

84 

69.17 

- 

69.99 

85.83 

86.66 

83 

68.33 

- 

69.16 

85.00 

- 

85.82 

82 

67.50 

- 

68.32 

84.17 

- 

84.99 

81 

66.67 

- 

67.49 

83.33 

- 

84.16 

80 

65.83 

- 

66.66 

82.50 

- 

83.32 

79 

65.00 

- 

65.82 

Below 

65 

42 


TABLE  22 


ATC  ACADEMY  STUDENTS  FROM  CLASSES  78-01,  78-02,  9001  to  9006 
FAILING  PRIMARY  POSITION  SKILLS  TEST  OR  LABORATORY  PROBLEMS 

OR 

FINAL  PHASE  GRADE 


Skills  Test 

Laboratory 

New  Phas 

Student 

PF 

IF 

ES 

PF 

IF  BS 

Grade 

78-01 

1 

69 

9 

68 

78-02 

1 

34 

8 

10 

35 

9001 

2 

35 

5 

69 

3 

33 

4 

10 

6 

33 

67.3 

8 

9 

11 

10 

12 

70 

13 

8 

33 

70.3 

65 

14 

9 

16 

67.2 

9002 

1 

10 

11 

34 

9003 

2 

6 

4 

70.25 

69 

9004 

1 

67.75 

3 

7 

10 

10 

68.50 

11 

13 

9005 

6 

71.5 

7 

75 

9 

75.5 

11 

7 

71.25 

13 

52.75 

69 

68 

14 

7 

15 

10 

14 

9006 

1 

75.75 

3 

35 

4 

10 

5 

6 

67.50 

71.00 

66 

7 

71.00 

10 

69.50 

12 

12 

56.50 

13 

14 

71.50 

_  - 

- 

7*" 

Totals 

12 

8 

7 

8 

6  7 

0 

Type 

Failure 


Phase  Grade 


Phase  Grade 


Phase  Grade 


Phase  Grade 


Posi tion 


Phase  Grade 


Position  & 
Phase  Grade 
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TABLE  23 

INTERCORRELATIONS  OF  PHASE  III  TESTS,  NEW  TRAINING  PROGRAM 
CLASSES  78-01  to  9006 
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